Camera module, image processing apparatus, and image processing method

ABSTRACT

According to one embodiment, a camera module includes a second imaging optical system and an image processing section. The second imaging optical system forms an image piece. The image processing section has at least one of an alignment adjustment section, a resolution restoration section, and a shading correction section and has a stitching section. The stitching section joins the image pieces, subjected to at least one of alignment adjustment, resolution restoration, and shading correction, together to form a subject image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2011-060155, filed on Mar. 18, 2011; theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a camera module, animage processing apparatus, and an image processing method.

BACKGROUND

Recently, there has been proposed a camera module having a compound-eyeconfiguration that can simultaneously photograph a subject from aplurality of viewpoints. In the camera module, by image processing of animage group photographed using the compound-eye configuration, a subjectdistance can be estimated, and a two-dimensional image can bereconstructed by joining images together, and the like. In the cameramodule, the depth information of a subject can be obtained from aplurality of images from different viewpoints. The camera moduleperforms image processing such as refocusing by utilizing the depthinformation.

As the compound-eye configuration of the camera module, there has beenknown, for example, one in which a sub-lens array is provided between animage sensor and a main lens through which light from a subject is takeninto an image sensor. The diameter of a sub lens constituting thesub-lens array is so small as approximately 140 μm, for example, and adistance between the sub-lens array and the image sensor is so short asapproximately 350 μm, for example. Consequently, a manufacturing errorand an installation error of the sub lens, the optical performance ofthe sub lens, and so on significantly affect the image quality. Thus, toobtain a high-quality image, the problems include reduction in yield ofthe camera module and increase in a manufacturing cost used forsuppressing the manufacturing error and the installation error of thesub lens.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of acamera module according to an embodiment;

FIG. 2 is a block diagram illustrating a configuration of a digitalcamera which is electronic equipment comprising the camera moduleillustrated in FIG. 1;

FIG. 3 is a cross-sectional schematic diagram of an imaging opticalsystem and an image sensor;

FIG. 4 is a schematic diagram of a light-incident-side flat surface ofthe image sensor;

FIG. 5 is an explanatory view of an image piece created by the imagesensor;

FIG. 6 is an explanatory view of a reconstruction processing of asubject image by the camera module;

FIG. 7 is a block diagram illustrating a constitution for imageprocessing in the camera module;

FIG. 8 is a flow chart illustrating a procedure of setting an alignmentadjustment correction coefficient for alignment adjustment in analignment adjustment section;

FIG. 9 is a view illustrating an example of an alignment adjustmentchart;

FIG. 10 is a flow chart illustrating a procedure of setting adeconvolution matrix for resolution restoration in a first resolutionrestoration section and a second resolution restoration section;

FIG. 11 is a flow chart illustrating a procedure of setting a shadingcorrection coefficient for shading correction in a first shadingcorrection section and a second shading correction section; and

FIG. 12 is a view illustrating an example of relative illuminance dataobtained for each lens.

DETAILED DESCRIPTION

In general, according to one embodiment, a camera module includes animaging section, a first imaging optical system, a second imagingoptical system, and an image processing section. The imaging sectioncomprises pixel cells arranged in the form of an array. The imagingsection images a subject image. The first imaging optical systemcomprises a main lens. The main lens takes light from a subject into theimaging section. The second imaging optical system is provided in anoptical path between the imaging section and the first imaging opticalsystem. The second imaging optical system forms an image piece. Theimage piece corresponds to a portion of the subject image. The secondimaging optical system forms the image piece for each pixel block. Thepixel block is constituted of a plurality of pixel cells. The imageprocessing section performs a signal processing of an image signal. Theimage signal is obtained by imaging the subject image in the imagingsection. The second imaging optical system forms the image piece by asub lens. The sub lens is provided corresponding to each of the pixelblocks. The image processing section has at least one of an alignmentadjustment section, a resolution restoration section, and a shadingcorrection section and has a stitching section. The alignment adjustmentsection performs alignment adjustment for correcting a deviation in animage piece due to an individual difference of the sub lens. Theresolution restoration section performs resolution restoration of theimage piece based on lens characteristics of the sub lens. The shadingcorrection section performs shading correction in the sub lens. Thestitching section joins the image pieces, subjected to at least one ofthe alignment adjustment, the resolution restoration, and the shadingcorrection, together to form the subject image.

Exemplary embodiments of a camera module, an image processing apparatus,and an image processing method will be explained below in detail withreference to the accompanying drawings. The present invention is notlimited to the following embodiments.

FIG. 1 is a block diagram illustrating a schematic configuration of acamera module according to the embodiment. FIG. 2 is a block diagramillustrating a configuration of a digital camera which is electronicequipment comprising the camera module illustrated in FIG. 1.

A digital camera 1 has a camera module 2, a storage section 3, and adisplay section 4. The camera module 2 images a subject image. Thestorage section 3 stores an image taken by the camera module 2. Thedisplay section 4 displays the image taken by the camera module 2. Thedisplay section 4 is a liquid crystal display, for example.

The camera module 2 outputs an image signal to the storage section 3 andthe display section 4 by imaging the subject image. The storage section3 outputs the image signal to the display section 4 in response touser's operation and so on. The display section 4 displays an image inresponse to the image signal input from the camera module 2 or thestorage section 3. An electronic equipment to which the camera module 2is applied may be a equipment other than the digital camera 1, such as acamera-equipped portable terminal.

The camera module 2 has an imaging module section 5 and an image signalprocessor (ISP) 6. The imaging module section 5 has an imaging opticalsystem 11, an image sensor 12, an imaging circuit 13, and an OTP (onetime programmable memory) 14. The imaging optical system 11 takes lightfrom a subject into the image sensor 12, and a subject image is formedby the image sensor 12. The image sensor 12 converts the light taken bythe imaging optical system 11 into a signal charge. The image sensor 12functions as an imaging section imaging the subject image.

The imaging circuit 13 drives the image sensor 12 and processes theimage signal from the image sensor 12. The imaging circuit 13 takesthereinto signal values of R (red), G (green), and B (blue) in ordercorresponding to a Bayer array and thereby generates an analog imagesignal. The imaging circuit 13 converts an obtained image signal from ananalog mode into a digital mode. The OTP 14 stores a parameter used forsignal processing of the image signal.

An ISP 6 has a camera module I/F (interface) 15, an image capturesection 16, a signal processing section 17, and a driver I/F (interface)18. A RAW image obtained by imaging in the imaging module section 5 iscaptured from the camera module I/F 15 by the image capture section 16.

The signal processing section 17 applies the signal processing to theRAW image captured by the image capture section 16. The driver I/F 18outputs the image signal subjected to the signal processing by thesignal processing section 17 to a display driver (not illustrated). Thedisplay driver displays the image imaged by the camera module 2.

FIG. 3 is a cross-sectional schematic diagram of an imaging opticalsystem and an image sensor. FIG. 4 is a schematic diagram of alight-incident-side flat surface of the image sensor. The image sensor12 comprises pixel cells 24 arranged in the form of an array. In theimage sensor 12, pixel blocks 25 constituted of a plurality of pixelcells 24 are set. The pixel block 25 is constituted of 25 pixel cells 24arranged in the form of an array of 5 by 5 in rows and columns, forexample.

The imaging optical system 11 has a main lens 21 and a sub-lens array22. The main lens 21 functions as a first imaging optical system takinglight from a subject into the image sensor 12. The sub-lens array 22 isprovided in an optical path between the image sensor 12 and the mainlens 21, such as an imaging plane of the main lens 12.

The sub-lens array 22 comprises sub lenses 23 arranged in the form of anarray. The sub lenses 23 are provided corresponding to each of the pixelblocks 25. Each of the sub lenses 23 forms as an image piece the subjectimage formed by the main lens 21. The image piece corresponds to aportion of the subject image. The sub-lens array 22 forms the imagepiece for each pixel block. The arrangement of the sub lenses 23 may beeither a square lattice arrangement or a hexagonal close-packedarrangement.

In the camera module 2, light entering the imaging optical system 11from the subject is divided by a plurality of the sub lenses 23, and theimage pieces as the same number as the sub lenses 23 are created in theimage sensor 12. The image sensor 12 generates a parallax according tothe arrangement position of the sub lenses 23 and thereby creates theimage pieces having information from different viewpoints.

FIG. 5 is an explanatory view of the image piece created by the imagesensor. FIG. 6 is an explanatory view of a reconstruction processing ofthe subject image by the camera module. In this example, a characterstring “ABCD” is imaged by the camera module 2, and the reconstructionprocessing is performed.

A field of view formed by each of the sub lenses 23 as the image piece26 has an overlap range corresponding to the parallax in the imagingplane of the main lens 21. As illustrated in FIG. 5, for example, thecharacter string “ABCD” is formed by the image sensor 12 so as to becomethe image pieces 26 whose overlapping portions are slightly differentfrom each other. Then, the camera module 2 joins the image pieces 26together so that the overlapping portions are coincided and therebyreconstructs the subject image. The image pieces 26 illustrated in FIG.5 is reconstructed to a subject image 27 including the character string“ABCD” as illustrated in FIG. 6 by such a signal processing that thecharacters “A”, “B”, “C”, and “D” are coincided with each other.

FIG. 7 is a block diagram illustrating a constitution for the imageprocessing in the camera module. The signal processing in the cameramodule 2 is roughly divided into the processing in the imaging modulesection 5 and the processing in the ISP 6. The imaging circuit 13 of theimaging module section 5 and the signal processing section 17 of the ISP6 function as an image processing section (image processing apparatus)performing the signal processing of the image signal obtained by imagingthe subject image in the image sensor 12.

The imaging circuit 13 has an alignment adjustment section 31, a firstresolution restoration section 32, and a first shading correctionsection 33. The alignment adjustment section 31, the first resolutionrestoration section 32, and the first shading correction section 33apply the signal processing to a RAW image 30 as a plurality of theimage pieces 26 obtained by imaging in the image sensor 12.

The alignment adjustment section 31 performs the alignment adjustment ofthe image pieces 26 for correcting the deviation of the image pieces 26due to the individual difference of the sub lens 23. The individualdifference of the sub lens 23 is a difference occurring for each of thesub lenses 23, such as the manufacturing error and the installationerror of the sub lens 23. The alignment adjustment section 31 performscoordinate transformation of the image pieces 26, using an alignmentadjustment correction coefficient previously stored in the OTP 14.

The first resolution restoration section 32 performs resolutionrestoration for each of the image pieces 26 based on the lenscharacteristics of the sub lens 23 (first resolution restoration). Asthe lens characteristics, a point spread function (PSF) is used, forexample. The first resolution restoration section 32 multiplies thedeconvolution matrix of the PSF, for example, and thereby restores animage with reduced blur. The deconvolution matrix of the PSF ispreviously stored in the OTP 14. The effect of the resolutionrestoration depends on algorithm used in the restoration. The firstresolution restoration section 32 uses the Richardson-Lucy method, forexample, in order to restore an image close to an original subjectimage.

The first shading correction section 33 performs the shading correctionfor correcting an illuminance unevenness caused by the sub lens 23, andparticularly a light quantity difference between a central portion ofthe image piece 26 and a peripheral portion thereof (first shadingcorrection). The first shading correction section 33 performs theshading correction of the image piece 26, using a shading correctioncoefficient previously stored in the OTP 14.

The signal processing section 17 has a second shading correction section44, a demosaicing section 41, a first scaling section 42, a stitchingsection 43, a noise reduction section 45, a second resolutionrestoration section 46, a crop section 47, and a second scaling section48.

The second shading correction section 44 performs the shading correctionfor correcting the illuminance unevenness caused by the main lens 21,and particularly the light quantity difference between a central portionof the subject image and a peripheral portion thereof (second shadingcorrection). The second shading correction section 44 performs theshading correction of the RAW image 30 subjected to the alignmentadjustment, the resolution restoration, and the shading correction inthe imaging circuit 13. The second shading correction section 44 usesthe shading correction coefficient previously stored in the OTP 14.

The demosaicing section 41 synthesizes a color bit map image by ademosaicing processing of the RAW image 30 subjected to the shadingcorrection in the second shading correction section 44. The firstscaling section 42 performs a scaling processing of the image piece 26.The stitching section 43 joins the image pieces 26 together to form abit map image 40 as a subject image. The noise reduction section 45removes noise of the subject image.

The second resolution restoration section 46 performs the resolutionrestoration of the bit map image 40 as the subject image based on thelens characteristics of the main lens 21 and the lens characteristics ofthe sub lens 23 (second resolution restoration). As with the firstresolution restoration section 32, the second resolution restorationsection 46 uses the deconvolution matrix of the PSF previously stored inthe OTP 14. Further, as with the first resolution restoration section32, the second resolution restoration section 46 uses theRichardson-Lucy method, for example.

The crop section 47 performs a crop processing of cutting a portion ofthe subject image. The second scaling section 48 performs a scalingprocessing of the subject image. The procedure of the processingdescribed in the present embodiment is an example, and the addition ofother processing, skip of processing that can be omitted, and a changeof the processing order may be suitably performed. The signal processingby each component may be performed by either the imaging circuit 13 orthe signal processing section 17 or may be performed to be shared byboth of them. For example, the alignment adjustment section 31, thefirst resolution restoration section 32, and the first shadingcorrection section 33 may be provided in the signal processing section17.

FIG. 8 is a flow chart illustrating a procedure of setting the alignmentadjustment correction coefficient for the alignment adjustment in thealignment adjustment section. The alignment adjustment correctioncoefficient is set in the manufacturing process of the camera module 2,for example. In step S11, a chart for alignment adjustment is arranged,and the alignment adjustment chart is taken by the camera module 2.

FIG. 9 is a view illustrating an example of the alignment adjustmentchart. A plurality of adjustment markers 51 are described in analignment adjustment chart 50. The adjustment markers 51 are arranged inthe form of a matrix of five in the longitudinal direction and five inthe lateral direction. The number of the adjustment markers 51 in thealignment adjustment chart 50 may be suitably changed.

The adjustment marker 51 is a mark formed by connecting corners of twoblack squares with each other, and the position at which the corners areconnected with each other is the coordinate of the adjustment marker 51.The adjustment marker 51 may have any shape as long as it can specifythe position on the alignment adjustment chart 50. The arrangement ofthe adjustment markers 51 may be suitably changed. For example, whenthere is a range in which high-definition photographing is especiallydesired, a large number of the adjustment markers 51 may be arranged inthe range.

In step S12, the obtained image pieces 26 are made to be a singlesubject image in the stitching section 43, and an image constituted of asignal of G among R, G, and B (suitably referred to as a “G image”) isgenerated. In the image sensor 12, with regard to a pixel for R and apixel for B, a signal value of a surrounding pixel for G isinterpolated, whereby a signal value of G is generated. In the case ofphotographing at a low illuminance, or in the case where the sensitivityof the image sensor 12 is low, a G image may be generated after thenoise reduction.

In step S13, each coordinate of the adjustment markers 51 is calculatedfrom the G image generated in step S12. In step S14, the alignmentadjustment correction coefficient is calculated from the coordinate ofthe adjustment marker 51 calculated in step S13. In step S15, thealignment adjustment correction coefficient calculated in step S14 iswritten in the OTP 14.

The alignment adjustment correction coefficient is a coefficient inmatrix operation. The alignment adjustment correction coefficient isobtained by the following formulae, using a least-squares method, forexample:Y=kXK=YX ^(t) [XX ^(t)]⁻¹,

wherein k is the alignment adjustment correction coefficient, Y is thecoordinate of the adjustment marker 51 calculated in step S13, and X isa coordinate previously set as standard. X^(t) is a transposed matrix ofX. [XX^(t)]⁻¹ is an inverse matrix of XX^(t). Although the alignmentadjustment correction coefficient is obtained by the least-squaresmethod, it may be obtained using other algorithm such as a nonlinearoptimization method.

The alignment adjustment section 31 reads the alignment adjustmentcorrection coefficient from the OTP 14 for each photographing by thecamera module 2. Further, the alignment adjustment section 31 appliescoordinate conversion using the alignment adjustment correctioncoefficient read from the OTP 14 to the RAW image 30 obtained by theimage sensor 12.

The alignment adjustment section 31 performs the coordinate conversionby calculation using the following formula, for example. k_(ij) is thealignment adjustment correction coefficient, (x, y) is a coordinatebefore correction, and (x′, y′) is a coordinate after the correction.

$\begin{bmatrix}x^{\prime} \\y^{\prime} \\1\end{bmatrix} = {\begin{bmatrix}k_{11} & k_{12} & k_{13} \\k_{21} & k_{22} & k_{23} \\0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}x \\y \\1\end{bmatrix}}$

The camera module 2 can suppress the deviation in the image piece 26 dueto the manufacturing error and the installation error of the sub lens 23by the coordinate conversion in the alignment adjustment section 31.Although the alignment adjustment section 31 performs the coordinateconversion collectively by the matrix operation, the coordinateconversion may be performed portion by portion by the calculation usingthe alignment adjustment correction coefficient suitably changedaccording to an image height. When a vertical axis perpendicular to anoptical axis of a lens is assumed, the image height is a distance alongthe vertical axis from an intersection between the vertical axis and theoptical axis.

The alignment adjustment section 31 may perform the coordinateconversion by referring to a lookup table, for example, instead of thematrix operation. The alignment adjustment correction coefficient maynot be calculated after the generation of the G image from the RAWimage. The alignment adjustment correction coefficient may be calculatedbased on the G image extracted from the color bit map image, forexample.

FIG. 10 is a flow chart illustrating a procedure of setting thedeconvolution matrix for the resolution restoration in the firstresolution restoration section and the second resolution restoration.The deconvolution matrix may be set in the manufacturing process of thecamera module 2, for example. In step S21, a test chart is taken by thecamera module 2, and photographing data obtained by photographing iscalculated, whereby PSF data is obtained. The PSF data is obtained by,for example, presuming a standard image free from blur and measuring thelevel of blur of an observation image relative to the standard image. Inthe test chart, the imaging plane of the image sensor 12 is virtuallydivided into nine areas of three rows by three columns, for example, andthe test chart is a point image chart constituted of a plurality ofpoint images.

In step S22, the deconvolution matrix for each image height iscalculated based on the PSF data obtained in step S21. The deconvolutionmatrix with respect to the subject image reconstructed from the imagepieces 26 and the deconvolution matrix with respect to each of the imagepieces 26 are calculated. The deconvolution matrix with respect to theimage piece 26 reflects the PSF data for each image height of the sublens 23. The deconvolution matrix with respect to the subject imagereflects the PSF data for each image height of the main lens 21 and thePSF data grouped for each of the sub lenses 23. In step S23, thedeconvolution matrix calculated in step S22 is written in the OTP 14.

The first resolution restoration section 32 reads the deconvolutionmatrix with respect to the image piece 26 from the OTP 14 for eachtaking by the camera module 2. The first resolution restoration section32 multiplies the deconvolution matrix for each image height of the sublens 23 by the RAW data of the image piece 26.

The second resolution restoration section 46 reads the deconvolutionmatrix with respect to the subject image from the OTP 14 for each takingby the camera module 2. Further, the second resolution restorationsection 46 multiplies the deconvolution matrix for each image height ofthe main lens 21 and the deconvolution matrix for each of the sub lenses23 by the bit map data of the subject image.

The resolution restoration method by multiplying the deconvolutionmatrix is based on such a theory that the observation image can beexpressed by a real image and convolution of a PSF function causingdeterioration of an image. In the camera module 2, by virtue of themultiplication of the deconvolution matrix in the first resolutionrestoration section 32, the blur for each of the image pieces 26 due tothe lens characteristics of the sub lens 23 can be suppressed.

In the camera module 2, by virtue of the multiplication of thedeconvolution matrix in the second resolution restoration section 46,the blur of the subject image due to the lens characteristics of themain lens 21 and the lens characteristics of the sub lens 23 can besuppressed. At least one of the first resolution restoration section 32and the second resolution restoration section 46 may perform the dataconversion by referring to the lookup table, for example, instead of thematrix operation.

FIG. 11 is a flow chart illustrating a procedure of setting the shadingcorrection coefficient for the shading correction in the first shadingcorrection section and the second shading correction section. Theshading correction coefficient is set in the manufacturing process ofthe camera module 2, for example. In step S31, relative illuminance dataof the main lens 21 is obtained. In step S32, relative illuminance dataof each of the sub lenses 23 is obtained. The order of step S31 and stepS32 is arbitrary.

FIG. 12 is a view illustrating an example of the relative illuminancedata obtained for each lens. The relative illuminance data is therelative illuminance for each image height. The relative illuminance isa relative illuminance in a case where the illuminance at the imageheight of zero is 100%. Usually, the relative illuminance of a lens isreduced as the image height is increased.

In step S33, the shading correction coefficient in the main lens 21 andthe shading correction coefficient in the sub lens 23 are obtained to bewritten in the OTP 14. As the shading correction coefficient in the mainlens 21, such a coefficient that offsets an illuminance difference foreach image height of the subject image is set based on the relativeilluminance data obtained in step S31. As the shading correctioncoefficient in the sub lens 23, such a coefficient that offsets theilluminance difference for each image height of the image piece 26 isset for each of the sub lenses 23 based on the relative illuminance dataobtained in step S32.

The first shading correction section 33 reads the shading correctioncoefficient in each of the sub lenses 23 from the OTP 14 for each takingby the camera module 2. Further, the first shading correction section 33multiplies the RAW data of each of the image pieces 26 by the shadingcorrection coefficient of the sub lens 23 corresponding to each of them.

The second shading correction section 44 reads the shading correctioncoefficient in the main lens 21 from the OTP 14 for each taking by thecamera module 2. Further, the second shading correction section 44multiplies the bit map data of the subject image by the shadingcorrection coefficient in the main lens 21.

By virtue of the multiplication of the shading correction coefficient inthe first shading correction section 33, the camera module 2 cansuppress the illuminance unevenness of each of the image pieces 26caused by the sub lens 23. Further, by virtue of the multiplication ofthe shading correction coefficient in the second shading correctionsection 44, the camera module 2 can suppress the illuminance unevennessof the subject image caused by the main lens 21.

At least one of the first shading correction section 33 and the secondshading correction section 44 may perform the data conversion byreferring to the lookup table, for example, instead of themultiplication of the shading correction coefficient.

The camera module 2 comprises the alignment adjustment section 31, thefirst resolution restoration section 32, and the first shadingcorrection section 33, whereby the influences of the manufacturing errorand the installation error of the sub lens 23, the optical performanceof the sub lens 23, and so on the image quality are suppressed.Consequently, in the camera module 2, a high-quality image can beobtained using a compound-eye configuration that can simultaneouslyphotograph the same subject from a plurality of viewpoints.

The camera module 2 may not comprise all of the alignment adjustmentsection 31, the first resolution restoration section 32, and the firstshading correction section 33. The camera module 2 may comprise at leastone of the alignment adjustment section 31, the first resolutionrestoration section 32, and the first shading correction section 33.Consequently, the camera module 2 can suppress at least one of theinfluences such as the individual difference of the sub lens 23 and theoptical performance thereof, and a good image quality can be obtained.

The camera module according to the embodiment may be applied toelectronic equipment other than a digital camera, such as acamera-equipped cell-phone.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. A camera module comprising: an imaging sectionwhich comprises pixel cells arranged in a form of an array and images asubject image; a first imaging optical system which comprises a mainlens through which light from a subject is taken into the imagingsection; a second imaging optical system which is provided in an opticalpath between the imaging section and the first imaging optical systemand forms an image piece, corresponding to a portion of the subjectimage, for each pixel block constituted of a plurality of the pixelcells; and an image processing section which performs a signalprocessing of an image signal obtained by imaging the subject image inthe imaging section, wherein the second imaging optical system forms theimage piece by a sub-lens provided corresponding to each of the pixelblocks, and the image processing section includes a first shadingcorrection section performing shading correction for correcting anilluminance unevenness caused by the sub-lens; a stitching section whichjoins the image pieces, subjected to the shading correction by the firstshading correction section, together to form the subject image; and asecond shading correction section performing shading correction of thesubject image formed by joining the image pieces together in thestitching section for correcting an illuminance unevenness caused by themain lens.
 2. The camera module according to claim 1, wherein the imageprocessing section further includes at least one of: an alignmentadjustment section performing alignment adjustment for correcting adeviation in the image piece due to an individual difference of thesub-lens; and a resolution restoration section performing resolutionrestoration of the image piece based on lens characteristics of thesub-lens, and the stitching section joins the image pieces, subjected toat least one of the alignment adjustment and the resolution restoration,together to form the subject image.
 3. A camera module comprising: animaging section which comprises pixel cells arranged in a form of anarray and images a subject image; a first imaging optical system whichcomprises a main lens through which light from a subject is taken intothe imaging section; a second imaging optical system which is providedin an optical path between the imaging section and the first imagingoptical system and forms an image piece, corresponding to a portion ofthe subject image, for each pixel block constituted of a plurality ofthe pixel cells; and an image processing section which performs a signalprocessing of an image signal obtained by imaging the subject image inthe imaging section, wherein the second imaging optical system forms theimage piece by a sub-lens provided corresponding to each of the pixelblocks, and the image processing section includes a first resolutionrestoration section performing resolution restoration of the image piecebased on lens characteristics of the sub-lens; a stitching section whichjoins the image pieces, subjected to the resolution restoration by thefirst resolution restoration section, together to form the subjectimage; and a second resolution restoration section performing resolutionrestoration of the subject image formed by joining the image piecestogether in the stitching section based on lens characteristics of themain lens and the lens characteristics of the sub lens.
 4. The cameramodule according to claim 3, wherein the image processing sectionfurther includes at least one of: an alignment adjustment sectionperforming alignment adjustment for correcting a deviation in the imagepiece due to an individual difference of the sub-lens; and a shadingcorrection section performing shading correction in the sub-lens, andthe stitching section joins the image pieces, subjected to at least oneof the alignment adjustment and the shading correction, together to formthe subject image.
 5. An image processing apparatus, which performs asignal processing of an image signal obtained by imaging a subject imagein an imaging section, comprising: a first resolution restorationsection performing resolution restoration of an image piece based onlens characteristics of a sub-lens for each pixel block, the image piecebeing formed by the sub-lens corresponding to the pixel block, the pixelblock being constituted of a plurality of pixel cells arranged in a formof an array in the imaging section; a stitching section which joins theimage pieces, subjected to the resolution restoration by the firstresolution restoration section, together to form the subject image; anda second resolution restoration section performing resolutionrestoration of the subject image formed by joining the image piecestogether in the stitching section based on lens characteristics of amain lens and the lens characteristics of the sub-lens, the main lensthrough which light from a subject is taken into the imaging section. 6.The image processing apparatus according to claim 5 further comprisingat least one of: an alignment adjustment section performing alignmentadjustment for correcting a deviation in the image piece due to anindividual difference of the sub-lens; and a shading correction sectionperforming shading correction in the sub-lens, and the stitching sectionjoins the image pieces, subjected to at least one of the alignmentadjustment and the shading correction, together to form the subjectimage.